{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "50110eec",
   "metadata": {},
   "outputs": [],
   "source": [
    "%run -i \"../util/file_utils.ipynb\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "d02a0f68",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "To Sherlock Holmes she is always _the_ woman. I have seldom heard him\n",
      "mention her under any other name. In his eyes she eclipses and\n",
      "predominates the whole of her sex. It was not that he felt any emotion\n",
      "akin to love for Irene Adler. All emotions, and that one particularly,\n",
      "were abhorrent to his cold, precise but admirably balanced mind. He\n",
      "was, I take it, the most perfect reasoning and observing machine that\n",
      "the world has seen, but as a lover he would have placed himself in a\n",
      "false position. He never spoke of the softer passions, save with a gibe\n",
      "and a sneer. They were admirable things for the observer—excellent for\n",
      "drawing the veil from men’s motives and actions. But for the trained\n",
      "reasoner to admit such intrusions into his own delicate and finely\n",
      "adjusted temperament was to introduce a distracting factor which might\n",
      "throw a doubt upon all his mental results. Grit in a sensitive\n",
      "instrument, or a crack in one of his own high-power lenses, would not\n",
      "be more disturbing than a strong emotion in a nature such as his. And\n",
      "yet there was but one woman to him, and that woman was the late Irene\n",
      "Adler, of dubious and questionable memory.\n"
     ]
    }
   ],
   "source": [
    "sherlock_holmes_part_of_text = read_text_file(\"../data/sherlock_holmes_1.txt\")\n",
    "print(sherlock_holmes_part_of_text)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3f22146f",
   "metadata": {},
   "source": [
    "# Divide into sentences using nltk"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "beaf9a8c",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['To Sherlock Holmes she is always _the_ woman.', 'I have seldom heard him\\nmention her under any other name.', 'In his eyes she eclipses and\\npredominates the whole of her sex.', 'It was not that he felt any emotion\\nakin to love for Irene Adler.', 'All emotions, and that one particularly,\\nwere abhorrent to his cold, precise but admirably balanced mind.', 'He\\nwas, I take it, the most perfect reasoning and observing machine that\\nthe world has seen, but as a lover he would have placed himself in a\\nfalse position.', 'He never spoke of the softer passions, save with a gibe\\nand a sneer.', 'They were admirable things for the observer—excellent for\\ndrawing the veil from men’s motives and actions.', 'But for the trained\\nreasoner to admit such intrusions into his own delicate and finely\\nadjusted temperament was to introduce a distracting factor which might\\nthrow a doubt upon all his mental results.', 'Grit in a sensitive\\ninstrument, or a crack in one of his own high-power lenses, would not\\nbe more disturbing than a strong emotion in a nature such as his.', 'And\\nyet there was but one woman to him, and that woman was the late Irene\\nAdler, of dubious and questionable memory.']\n",
      "11\n"
     ]
    }
   ],
   "source": [
    "import nltk\n",
    "#nltk.download('punkt') # Run this line only the first time you run this notebook to download tokenizer data\n",
    "tokenizer = nltk.data.load(\"tokenizers/punkt/english.pickle\")\n",
    "sentences_nltk = tokenizer.tokenize(sherlock_holmes_part_of_text)\n",
    "print(sentences_nltk)\n",
    "print(len(sentences_nltk))"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8adc9ca6",
   "metadata": {},
   "source": [
    "# Divide into sentences using spacy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "264dc36b",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['To Sherlock Holmes she is always _the_ woman.', 'I have seldom heard him\\nmention her under any other name.', 'In his eyes she eclipses and\\npredominates the whole of her sex.', 'It was not that he felt any emotion\\nakin to love for Irene Adler.', 'All emotions, and that one particularly,\\nwere abhorrent to his cold, precise but admirably balanced mind.', 'He\\nwas, I take it, the most perfect reasoning and observing machine that\\nthe world has seen, but as a lover he would have placed himself in a\\nfalse position.', 'He never spoke of the softer passions, save with a gibe\\nand a sneer.', 'They were admirable things for the observer—excellent for\\ndrawing the veil from men’s motives and actions.', 'But for the trained\\nreasoner to admit such intrusions into his own delicate and finely\\nadjusted temperament was to introduce a distracting factor which might\\nthrow a doubt upon all his mental results.', 'Grit in a sensitive\\ninstrument, or a crack in one of his own high-power lenses, would not\\nbe more disturbing than a strong emotion in a nature such as his.', 'And\\nyet there was but one woman to him, and that woman was the late Irene\\nAdler, of dubious and questionable memory.']\n",
      "11\n"
     ]
    }
   ],
   "source": [
    "import spacy\n",
    "#!python -m spacy download en_core_web_sm # Run this line only the first time you run this notebook to download the model\n",
    "nlp = spacy.load(\"en_core_web_sm\")\n",
    "doc = nlp(sherlock_holmes_part_of_text)\n",
    "sentences_spacy = [sentence.text for sentence in doc.sents]\n",
    "print(sentences_spacy)\n",
    "print(len(sentences_spacy))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "34d8a0b5",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "True\n"
     ]
    }
   ],
   "source": [
    "print(sentences_nltk == sentences_spacy)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ab232fac",
   "metadata": {},
   "source": [
    "# Compare the time it takes using both methods"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "9ab509ed",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "NLTK: 0.0007140636444091797 s\n",
      "spaCy: 0.05654287338256836 s\n"
     ]
    }
   ],
   "source": [
    "import time\n",
    "\n",
    "def split_into_sentences_nltk(text):\n",
    "    sentences = tokenizer.tokenize(text)\n",
    "    return sentences\n",
    "\n",
    "def split_into_sentences_spacy(text):\n",
    "    doc = nlp(text)\n",
    "    sentences = [sentence.text for sentence in doc.sents]\n",
    "    return sentences\n",
    "\n",
    "start = time.time()\n",
    "split_into_sentences_nltk(sherlock_holmes_part_of_text)\n",
    "print(f\"NLTK: {time.time() - start} s\")\n",
    "\n",
    "start = time.time()\n",
    "split_into_sentences_spacy(sherlock_holmes_part_of_text)\n",
    "print(f\"spaCy: {time.time() - start} s\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c50e992f-2e90-43c2-92aa-9a7114b72bb0",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.12"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
